The volume of data required for running a business or an organization grows quickly and continuously. It stresses the technical infrastructure and worsens financial burdens, especially when data streams have to be sent from one source to many destinations, across local area networks (LAN's) and/or wide area networks (WAN's). Sending data streams using broadcast or multicast data delivery techniques can reduce the loading on data sources and the networks, but the use of data compression is still necessary to maximize the use of network capacity. Compression using dynamically generated dictionaries can improve data compression ratios of data with complex data types, frequent data changes or/and data values without obvious boundaries. Most dictionary-based compression schemes require lossless communication paths because both the transmitter and the receiver have to derive dictionaries on-the-fly from the input data. This requires precise synchronization between the dictionary generation in the transmitter and the dictionary generation in the receiver. However, unrecoverable data loss in the network will break the synchronization of dictionary generation between the transmitter and the receiver. The communication will then break down until the compression is reset. In one-to-many broadcast or multicast data streaming, data loss in one destination will impact all destinations. Moreover, the reset is either technically infeasible, functionally complicated, or bandwidth expensive.
There are many well-developed high performance compression dictionary-based algorithms which are designed for session-based or reliable communication channels, e.g. those algorithms used by winzip or pkzip. A dynamic dictionary will be created by the compressor and de-compressor simultaneously to cope with the data pattern changes. Both the compressor and de-compressor have to use the same dictionary harvesting algorithm which will determine the compression efficiency.
In one-way satellite or unreliable delivery channels without feedback paths, loss of one single data packet will cause the building of a dictionary in the de-compressor to be out-of-sync with building of dictionaries in the compressor. The communications between the compressor and the decompressor will break down as a result.